Add interactive text-query annotation button#1708
Merged
Merged
Conversation
7858f2f to
20303f0
Compare
37bc67d to
fc8e213
Compare
SAM3 text-query button, dialog, API, and IPC handlers, on top of the merged interactive segmentation + stereo work on main. The button lives in the annotation type list (with the 't' shortcut) alongside rectangle/polygon/line/segment, and opens a dialog to run a SAM3 text query on the current frame or, optionally, across all frames as a pipeline job. Desktop-only: gated behind a textQueryEnabled prop set only by the desktop ViewerLoader, so the web client renders nothing. Brings the feature to parity with origin/viame/master (SAM3 attribution text, persistent dialog while loading, exit edit mode on completion).
fc8e213 to
25ad49b
Compare
Route the single-frame text query through the correctly-named text-query-submit event so the submit handler actually runs and surfaces failures instead of silently doing nothing. Resolve the frame's media path and frame time via the segmentation resolvers so text query works on video and multicam datasets, not just image sequences.
Adds an off-by-default 'Replace existing annotations' toggle to the text query dialog. On the current-frame path it clears the current frame's annotations before applying the results; on the all-frames path it forwards replace_existing to the SAM3 refiner pipeline. Also corrects the all-frames pipeline file name and adds the refiner 'sam3:' param prefixes so the query text and threshold actually reach the refiner.
The interactive (current-frame) text query ran with no feedback while the model processed the frame. Add a progress dialog shown for the duration of the single-frame query; the all-frames path is unaffected since it runs as a pipeline job with its own progress UI.
Single-frame text query created every track with the default confidence of 1.0 because TrackStore.add() seeds the confidence pair at 1.0 and the handler never applied the detection's score. Call setType with det.score so the model's confidence is reflected instead of always showing 100%.
When a job process exits non-zero (and was not cancelled), scan its captured output for lines beginning with 'ERROR:' and show them in a dialog. Processes emit a single 'ERROR: <cause>' line to report the real failure reason (e.g. a GPU out-of-memory during SAM3 model load), so the user sees the cause without opening the job log.
021d883 to
4ddb7b1
Compare
mattdawkins
added a commit
that referenced
this pull request
Jul 1, 2026
Text queries replace existing annotations by default instead of accumulating; users can still uncheck it in the dialog to add alongside existing results.
Opening the text-query dialog and running a text query previously called initialize(), which warms the point-segmentation model. Text query uses its own (often different/heavier) model, so start the service process only (ensureStarted) and let the text-query model load lazily on first query.
runTextQueryPipeline passed the refiner settings as a flat pipelineParams object, but runPipeline() only threads pipelineParams.kwiverParams into the runner's -s flags -- so text_query/replace_existing/detection_threshold were silently dropped and the refiner ran with its defaults (querying 'object', not the user's text). Nest them under kwiverParams.
BryonLewis
approved these changes
Jul 3, 2026
BryonLewis
left a comment
Collaborator
There was a problem hiding this comment.
Was able to test this morning and I could get through the workflow properly.
In the future I may want to start dividing some other stuff up to reduce ViewerLoader and EditorMenu to make them smaller but it isn't needed for this PR.
I made a small change to add the confidence number to the label because only being seen in the thumb label when moving didn't seem right.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Follow-up to #1582. Adds the SAM3 interactive text-query annotation button on top of the interactive segmentation + stereo branch.
Changes
textQuery/refineDetections/runTextQueryPipelinefrontend API + IPC handlers.text-query-init/text-query/text-query-all-frames, service-ready forwarding).Notes
dev/add-interactive-seg-and-stereo; review/merge Add interactive segmentation, stereo, and multi-polygon support #1582 first.